Antje Hofmann: Learning Syntactic Relations with the Mole Task (2)
RePsychLing in SMLP2022
Author
Reinhold Kliegl
1 Background
This version removes the outlier subject “p10” found in the version 1.
Children’s (age: 4-8 years)reaction times in a task teaching them syntactic relations.
1.1 Overview
Original analysis is by Antje Hofmann.
MixedModels.jl version
Addition of new chunks illustrate
selection of parsimonious LMM using random-effects PCA
plotting conditional means
illustration of borrowing strength
2 Readme
2.1 Variables
Subj: Participant ID (renamed from ID; random factor)
Item: Word ID (random factor)
age: 4 - 8 years
Block (within-Subj/within-Item):
1st Learning
2nd Learning
Disruption
Recovery
Target(renamend fom targetness)
non-syllable target
syllable target
rt: response time
3 Setup
3.1 Packages
First attach the MixedModels.jl package and other packages for plotting. The CairoMakie.jl package allows the Makie graphics system [@Danisch2021] to generate high quality static images. Activate that package with the SVG (Scalable Vector Graphics) backend.
Code
usingAlgebraOfGraphicsusingArrowusingCairoMakie # graphics back-endusingCategoricalArraysusingChainusingDataFramesusingDataFrameMacros # simplified dplyr-like data wranglingusingMixedModelsusingMixedModelsMakie # diagnostic plotsusingProgressMeterusingRandom # random number generatorsusingRCall # call R from JuliausingStatsModelsusingAlgebraOfGraphics: boxplotusingAlgebraOfGraphics: densityusingMixedModelsMakie: qqnormusingMixedModelsMakie: ridgeplotusingMixedModelsMakie: scatterusingMixedModelsMakie: caterpillarProgressMeter.ijulia_behavior(:clear);CairoMakie.activate!(; type="svg");
The data are available as an arrow file.
Most of preprocessing was done with R in RStudio (see Hofmann_Maulwurf.Rmd).
Order of factor levels should be checked.
Code
dat =DataFrame(Arrow.Table("./data/Hofmann_Maulwurf_rt.arrow"))transform!(dat,:Target => categorical =>:Target,:Block => categorical =>:Block,:age => (x -> x .-6) =>:a1, # center age at six years:rt => (x ->log.(x)) =>:lrt)describe(dat)dat =filter(row -> row.Subj !="p10", dat)
6,257 rows × 8 columns (omitted printing of 1 columns)
Subj
Item
age
Block
Target
rt
a1
String?
String?
Float64?
Cat…?
Cat…?
Float64?
Float64
1
p01
02SG.ogg
7.73
1st Learning
Non-target syllable
3285.0
1.73
2
p01
21SG.ogg
7.73
1st Learning
Non-target syllable
2873.0
1.73
3
p01
20SG.ogg
7.73
1st Learning
Non-target syllable
2169.0
1.73
4
p01
13SG.ogg
7.73
1st Learning
Non-target syllable
3133.0
1.73
5
p01
32SG.ogg
7.73
1st Learning
Non-target syllable
5793.0
1.73
6
p01
05SG.ogg
7.73
1st Learning
Non-target syllable
2064.0
1.73
7
p01
27SG.ogg
7.73
1st Learning
Non-target syllable
4546.0
1.73
8
p01
18SG.ogg
7.73
1st Learning
Non-target syllable
2949.0
1.73
9
p01
23SG.ogg
7.73
1st Learning
Non-target syllable
2957.0
1.73
10
p01
07SG.ogg
7.73
1st Learning
Non-target syllable
2508.0
1.73
11
p01
29SG.ogg
7.73
1st Learning
Non-target syllable
2232.0
1.73
12
p01
19SG.ogg
7.73
1st Learning
Non-target syllable
2517.0
1.73
13
p01
14SG.ogg
7.73
1st Learning
Non-target syllable
2234.0
1.73
14
p01
16SG.ogg
7.73
1st Learning
Non-target syllable
4272.0
1.73
15
p01
15SG.ogg
7.73
1st Learning
Non-target syllable
3389.0
1.73
16
p01
09SG.ogg
7.73
1st Learning
Non-target syllable
2510.0
1.73
17
p01
25SG.ogg
7.73
1st Learning
Non-target syllable
5079.0
1.73
18
p01
12SG.ogg
7.73
1st Learning
Non-target syllable
2622.0
1.73
19
p01
23PA.ogg
7.73
1st Learning
Target syllable
2161.0
1.73
20
p01
32PA.ogg
7.73
1st Learning
Target syllable
2141.0
1.73
21
p01
19PA.ogg
7.73
1st Learning
Target syllable
2311.0
1.73
22
p01
13PA.ogg
7.73
1st Learning
Target syllable
3049.0
1.73
23
p01
04PA.ogg
7.73
1st Learning
Target syllable
2175.0
1.73
24
p01
25PA.ogg
7.73
1st Learning
Target syllable
1750.0
1.73
25
p01
20PA.ogg
7.73
1st Learning
Target syllable
2158.0
1.73
26
p01
14PA.ogg
7.73
1st Learning
Target syllable
3274.0
1.73
27
p01
11PA.ogg
7.73
1st Learning
Target syllable
2434.0
1.73
28
p01
30PA.ogg
7.73
1st Learning
Target syllable
1859.0
1.73
29
p01
15PA.ogg
7.73
1st Learning
Target syllable
2351.0
1.73
30
p01
16PA.ogg
7.73
1st Learning
Target syllable
1853.0
1.73
⋮
⋮
⋮
⋮
⋮
⋮
⋮
⋮
Centering age at six years yields an interpretable GM
Factor levels can also be set when contrasts are defined (see below).
BoxCox check showed that reaction time rt [ms] should be transformed to speed [1/s] = [Hz]
Indicator variables for Target and Block generated in R.
4 LMM analysis
4.1 Contrasts
Code
contrasts =merge(Dict(:Target =>EffectsCoding()),Dict(:Block =>SeqDiffCoding()),Dict(nm =>Grouping() for nm in (:Subj, :Item)) );
LMM m_prm2 is a defensible solution according to \(\Delta\) AIC, \(\Delta\) BIC suggests we should not bother with CPs.
Code
coeftable(m_prm2)
Coef.
Std. Error
z
Pr(>
(Intercept)
7.69869
0.0371901
207.01
<1e-99
trgt
-0.0118293
0.00451408
-2.62
0.0088
trng
-0.070356
0.0262888
-2.68
0.0074
drpt
0.0457635
0.0231034
1.98
0.0476
rcvr
-0.042934
0.0224154
-1.92
0.0554
a1
-0.108394
0.0326348
-3.32
0.0009
trgt & trng
0.00212125
0.00949196
0.22
0.8232
trgt & drpt
-0.0370438
0.0150283
-2.46
0.0137
trgt & rcvr
0.00972961
0.0132374
0.74
0.4623
trgt & a1
-0.00701276
0.00399025
-1.76
0.0788
trng & a1
0.0279914
0.0230904
1.21
0.2254
drpt & a1
0.0273081
0.0201717
1.35
0.1758
rcvr & a1
-0.0167551
0.0195894
-0.86
0.3924
trgt & trng & a1
-0.00608894
0.00831708
-0.73
0.4641
trgt & drpt & a1
0.029943
0.0131747
2.27
0.0230
trgt & rcvr & a1
-0.00509192
0.011614
-0.44
0.6611
4.8 Diagnostic plots
Various visualizations are used to check whether or not data are defensibly modeled with an LMM. They may lead to removal of outliers, transformations of the dependent variable, and deliver valuable heuristic information to be followed up with exploratory post-hoc analyses or ideally replication of new insights gained this way. In practice, it appears that only severe violations will stop people from reporting a model.
4.8.1 Residuals over fitted
Code
scatter(fitted(m_prm2), residuals(m_prm2))
Looks like we missed some fast response times.
4.8.2 Q-Q plot
Code
qqnorm(m_prm2; qqline=:none)
Hm?
4.8.3 Residual distributions: observed vs. theoretical
Curves for residulas based on observed and theoretical values should correspond.
Code
let n =nrow(dat) dat_rz = (; value=vcat(residuals(m_prm2) ./std(residuals(m_prm2)), randn(n)), curve=repeat(["residual", "normal"]; inner=n), )draw(data(dat_rz) *mapping(:value; color=:curve) *density(; bandwidth=0.1); )end
Figure 1: Kernel density plot of the standardized residuals for model m1 versus a standard normal
They are a bit too narrow.
4.9 Conditional means of random effects
4.9.1 Subject-related conditional means of random effects
Figure 2: Prediction intervals on subject random effects for model m_prm2
4.9.2 Borrowing-strength plots
Shrinkage refers to the adjustment of subject-level or item-level predictions by taking population estimates into account. The further a subject’s/item’s estimate is from the fixed effect or the more variable or less reliable the subject’s/item’s estimate, the more the prediction will be shrunk towards the population estimate. Alternative terms for shrinkage are “borrowing strength” (Tukey) and regularization. My favorite is actually Tukey’s because indeed we borrow strength from the population estimates to make predictions for individual subjects’ effects. The goal of this section to illustrate the results of borrowing strength.
Subject-related conditional means of random effects revealed information about individual differences beyond fixed effects. Would these results also be visible in unconditional means, that is when we compute GM and experimental effects within subjects (i.e., as fixed effects) without borrowing strength from the population estimates?
In the following plots, effect estimates based on alone on each subject’s data (i.e., no pooling of data, no borrowing of strength) are plotted in pink and the subjects’ conditional means shown in the caterpillar plots are plotted in blue. The arrows indicate how much a subject’s prediction is changed by borrowing strength from knowledge of the population estimates.